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ABSTRACT 

The Course Attitude Questionnaire (CAQ) has been 
adainistered 100,000 times in 3,500 classes at the Pennsylvania State 
University since 1964 !;ith extensive use in modified form at ether 
institutions. The purpose of this research vas to investigate the 
reliability and internal validity of the CAQ and the temporal 
stability of its original factor structure. CAO responses vere 
obtained from 5,297 Penn State students in 272 classes during a 1973 
term. These data vere item^analyzed and submitted to 
principal-components and common*f actor analyses vith orthogonal and 
oblique transformations. Coefficient alpha reliabilities vere 
computed for the f actor*derived scales. The large normative data base 
vas also inspected. The factor structure vas not replicated by a 
"fit* of the derived factors to the CAQ-keyed subscales. Host factors 
vere highly correlated and their reliabilities appeared to be a 
function of redundant item content. A "general" dimension pervaded 
the instrument, accounting for much of the total and common^f actor 
variance. A trend tovard inflation and skevness in scores vas noted 
for the 10*year period. The CAQ) subscales should be interpreted vith 
caution and systematically revieved for currant validity. The CAQ, 
originally included in the Appendix of this document, has been 
removed because it does not reproduce. For related document see ED 
029508. (Author/RC) 



U.S. DEPARTMENT OF HEALTH. 
EDUCATION « WELFARC 
NATIONAL INSTITUTE OF 
EDUCATION 

THIS OOCUMENT HAS BEEN REPRO 
OUCEO EXACTLY AS RECElVEO FROM 
THE PERSON OR ORGANIZATION ORIGIN 

t ^ ATlNG »T POINTSOP VIEW OR OPINIONS 

^ STATEO DO NOT NECESSARILY REPRc 

SENTOFFICIAL NATIONAL INSTITUTEOF 
EOUCATlON POSITION OR POLICY 

oo 

CO 

o 



A I^GAOE WITH THE COURSE ATHTUDE QUESTIONNAIRE: 
A FACTORIAL STUDY 



Maurice W. Vlllano and Edward H. Rosenstocfc 
The Pennsylvania State Itolverslty 



CO 



CO 

o 



(The Course Attitude Questionnaire, originally included 
in the Appendix of this document, has been removed 
because it does not reproduce.) 

Paper presented at the Fourth Annual Convocation 
Northeastern Educational Research Association 
Ellenville» New York November 2, 1973 



ERIC 



A DECADE WITH THE COURSE ATTITUDE QUESTIONNAIRE: A FACTORIAL STUDY 

Ilaurlce W. Vlllano and Edvard H, Rosenstock 
The Pennsylvania State University 

Background 

The Course Attitude Questionnaire (CAQ) of R. E. Spencer has been administered 
over 100^000 times In almost 3^500 classes at the Pennsylvania State University 
since 1964. The Instrument was originally developed for instructional television 
but was later generalized to Include all aspects of instruction as a basis for 
course improvement .(Dick, 1965). The CAQ has also been used extensively in its 
present or modified form at other institutions and a large normative data base 
has been established. 

There has been a decline » however » in the use of the CAQ at the Pennsylvania 
State University. Over half of the major colleges or departments and several of 
the 13 Commonwealth branch campuses of the University now favor other rating 
forms adopted from nationally-known Instruments or constructed locally. 
Criticisms by students and faculty have alleged that the Course Attitude 
Questionnaire is ''too broad" for evaluative or diagnostic purposes » the item 
statements are repetitive or overlapping » and some areas are not represented. 

The feasibility of continued use of the CAQ on a Unlverslty-wlde basis has 
been under consideration. The purpose of this paper was to report an 
investigation of the current reliability and validity of the Course Attitude 
Questionnaire » and the temporal stability of its orlglnally-clalmed factor 
structure. A large sample of contemporary data was used in an attempt to 
replicate the earlier research of Dick (1965) and Spencer and Aleamonl (1969). 
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Research has not kept pace with the development and use of student evaluation 
Instruments (Bejar» 1973) and findings reported In the literature have been 
characterized as Inconsistent and often contradictory In their nature. The 
decrease In the use of rating forms » such as the CAQ» may be due to a lack of 
convincing validation data (Costln» Greenough & Menges» 1971). One of the most 
recent and extensive summaries In this area of concern was the First Invitational 
Conference on Faculty Effectiveness as Evaluated by Students held at Teople 
University earlier this year (Sockloff» 1973). Among some of the prominent 
discussants were J. Centra of ETS» P. Dressel and Warrington of Michigan State 
University^ W. McKeachle of the University of Michigan* and R* Perry and 
R. Baumann of the University of Toledo. A general consensus that emerged from 
this meeting reaffirmed the need for: (1) longitudinal studies; (2) Instruments 
that were observable* ratable* and acceptable; and (3) scoring and scaling systems 
that adequately discriminated among Instructors or courses. 

There has been a dearth of long-^range evaluations of some of the more* 
extensively used student rating forms. The findings of this investigation may 
therefore have some important educational implications. Despite the seeming 
limitations of Instruments like the CAQ* there is sufficient convergent evidence 
to suggest possible directions for their improvement. Minimally* there is a 
requirement for the periodic and systematic review of student evaluation devices 
to assure their current validity. 

Factorial studies* as in this investigation* have examined the reduced sets 
of dimensions that described and explained the variance of factors underlying 
student evaluations. Reliable factor-derived dimensions or clusters of items 
are sought in the analysis of the instrument and its sub~scales (Guertin & Bailey* 
1970) • Some approximate methods for comparing the temporal stability of factor 
structures are discussed by Nunnally (li67) and others. 
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Procedures 



The Instrument evaluated in this replication study was the Course Attitude 
Questionnaire. The CAQ can be described as having 50 forced-choice Llkert-type 
Items » Including 22 marched positive-negative Item-pairs » on a four-step Agree- 
Dlsagree scale (see Appendix). Negative items are reversed in scoring^ and the 
cumulatlve» uoveighted item responses are then summed to yield scores on each 
of six keyed sub-scales of General Course Attitude » Method » Content » Interest-* 
Attention^ Instructor, and "Other/' as well as a Total CAQ score. 

The response data source was provided by 7»031 Pennsylvania State University 
students administered the CAQ in 272 graduate and undergraduate classes during 
the Winter 1973 term. The sample was screened for missing item responses and 
reduced to S»297 Ss. 

The cumulative normative data compiled on the CAQ were revlev&l and comparisons 
of descriptive statistics were made at selected points during the past 10 years. 
The reliability coefficients for single-form internal consistency (split-half 
method) were calculated for groups of positive and negative items and between 
groups of half-positive and half-negative items (Nunnally, 1967). The estimated 
Coefficient Alpha reliabilities were computed for both the CAQ-*keyed and factor** 
derived sub-scales or item clusters (Cronbach» 1951). 

At the item level » the response data were evaluated by multiple modes of 
factor analysis as suggested by Glass and Taylor (1966). The generated 50 by 50 
in tercor relet ion matrix was submitted to a principal-components analysis » with 
unities in the diagonal, to replicate the derived dimensions and total test 
variance explained by the original CAQ factor structure (Hotelling, 1933). 
Some limitations of this approach are explained at the end of the section. The 
"R" matrix was also subjected to an Alpha factor analysis, with estimates of 



cosnmmalltles Inserted In the diagonal « to make Inferences regarding the 
generallsablllty of the derived factors to the CAQ Item content domain and to 
account for the conmon*f actor variance (Kaiser & Caffrey, 1965). A root criterion 
of unity was used for the extraction of factors In both analyses which were then 
submitted to orthogonal (Varlmax) rotations (Kaiser » 1958) and oblique (Obllmln) 
transformations (Barman, 1967). The most psychologically --meaningful and 
psychometrlcally«defenslble solutions were chosen for interpretation. 

This study was somewhat limited by the nature of the information that was 
available from the earlier investigations (Dick, 1965; Spencer « Aleamoni, 1969). 
For example, the factor loading matrix was not reported thus precluding direct 
comparison as in t^e computation of coefficients of congruence for pairs of 



factor vectors across the CAQ data samples (Rummel, 1970). The basis for the 



assignment of the dAQx items to the sub^^scales was brought into question (Dick, 



Listed below are the factors and the items which were assigned to 
each. The per cent of variance in the students* scores which is generally 
accounted for by each of the factors is also shown. These percentages are 
an approximate average which has been obtained from subsequent factor analyses 
oF the questionnaire (p. 4). /underlining supplle^/ 

Factorial invarlance was apparently hinted at as well as the "contrived^' structure 

of the "Other" sub-scale ( Ibid .): 

The factor analyses of the responses from various courses indicate that 
the factors identified above are fairly stable; with the exception of some 
of the items which are related to course content , the items generally cluster 
together as Indicated above . Some of tKe course content items occ ass tonally 
(sic) intercorrelate quite highly with the more general course attitude items 
XFactor 1). _ - - 



A final note should be made of the items in Factor 6 - specific or other 
items. Although these items did not inter-correlate with the other factors > 
they were retained in the present form of the questionnaire because of the 
specific Information which is available in the responses to these items . 
Although the respt^nses to these items are grouped together to form one factor 
0core» it Is possible to get the indlvidlual distributions of*responses to ^ 
these and other items on the questionnaire (pp. 7-8). /underlining supplied/ 
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Findings 

There was a dlscern.ible Inflationary trend In Course Attitude Questionnaire 
ratings over the past decade. General Increases in the CAQ sub-scale and total 
scores were noted for Che time Intervals indicated (see Tabic 1). An inspection 
of the cumulative norms through 1973 revealed there was not a sufficient spread 
in the class means for the courses to discriminate adequately except at the 
extreme points (10th- and 90th-percentlles) of the distribution (see Tabic 2). 

The substantial reliabilities (.SO's and •90's) for the CAQ-deflned or keyed 
sub-scales and the overall Instrument that had been reported in the earlier studies 
were confirmed (see Table 3) • The magnitude of these reliability coefficients was 
due in part to the possibly spurious effect of the redundant item-pairs that con- 
tributed to the high Intercorrelatlons of items within scales. 

The obliquely-rotated seven-factor soulutlons for both the principal-components 
and Alpha factor analyses were determined to most closely approximate the purported 
CAQ factor structure. The results of the two modes factor analysis were similar 
except for minor differences in the factor assignments of a few items having 'Hreak" 
loadings (4» 7» 32» and 42); also> the loading coefficients of the Alpha factors 
were generally lower (see Tables 4 and 5) • The coefficients of congruence accross 
both factoring methods approached unity signifying almost perfect agreement. 

The empirically-derived factor scales did not provide a "close fit" to the 

a priori or CAQ-deflned or keyed sub-scales. The artificiality"- of the "Other-1- 

sub-scale was verified by its fission luto two factors (V and VII). One-fourth 
of the items loaded highest on "non-relevant" factors. Several items exhibited 
a factorial, complexity of two or more. Some items failed to achieve loading 
coefflclexnc^ of .40 or greater on any factor. 

ERIC 



6 

Sone weak Alpha factors suggested that the sampling from the content 
universe of CAQ Items may not have been complete. Fever Items clustered on 
the derived factors that resembled the Content » Int erest*At tent Ion » and ^Other" 
8ub--8cale8 as compared to those for General Course Attitude » Method » and 
Instructor (see Table 5). The reliabilities for the former group were also 
somewhat lower (see Table 6)* For these factors » at least » a reliable Inference 
could not be made with respect to their "positive generalizabillty" to the 
content domain from which the items had been drawn. The shift of items from 
their original CAQ^keyed assignments had been observed (see Table 7). 

A "general" dimension appeared to pervade the overall CAQ instrument. The 
first extracted factor in both analyses accounted alone for 72~percent of the 
common variance and 43~percent of the total variance. Furthermore » the primary 
axes of the factors corresponding to the General Course Attitude^ !fethod» and 
Interest-Attention sub-^scales were highly Intercorrelated (above « 50) (see 
Table 8), It was condtxded that the factorial Invariance of the Course Attitude 
Questionnaire had not been demonstrated. It was recommended that scores derived 
from the CAQ sub^scales be Interpreted with caution and that research on the 
relationship of the CAQ with important educational variables proceed with care« 

It was further recommended that any revised version of the CAQ be patterned 
after the model of the current Illinois Course Evaluation QuesCionaaire (CEQ 

Form 72) where duplicative It^^ bay^._baen eltalnated or 9P^ i^§A 

items incorporated for specific diagnostics » and provision made for open-ended 
or free-response comnents by students (Aleamonit 1973). 
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TABLE 2 

CUMULATIVE NORMATIVE DATA FOR THE COURSE ATTITUDE QUESTIONNAIRE 
AT THE PENNSYLVANIA STATE UNIVERSITY THROUGH SUMMER TERM 1973 









Percentile 










Sub -scale 


10 


25 


50 


75 


90 


X 


SD 


Gen Crse 
Attitude 


21.7 


22.9 


25.2 


26.8 


28.0 


24.7 


2.5 


Method 


19.2 


20.8 


23.0 


25.0 


27.0 


22.9 


3.0 


Content 


20.6 


22.2 


23.3 


24.7 


25.8 


23.1 


2.0 


Interest- 
Attention 


18.6 


20.9 


23.0 


24.6 


26.1 


22.7 


3.0 


Instructor 


22,7 


24.1 


25.2 


27.0 


28.1 


25.4 


2.1 


Other 


26. 6 


27.6 


29.4 


30.0 


31.8 


29.0 


2.1 


Total 


129.8 


139.5 


148.6 


156.9 


165.3 


147.8 


13.8 



o 
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TABLE 3A 

SPLIT-HALF RELIABILITIES FOR THE CAQ 



Groups 
of Items 




1969 Study ^ 
(N = 297) 


1973 Study 
(N = 5,297) 


22 positive versus 
22 negative matched- 
pair items 




.92 


.92 


50 items in two groups 
of half -positive and 
half-negative Items 




.93 


.91 


^Spencer & Aleamoni (1969, pp. 


7-8) 








TABLE 3B 




COEFFICIENT ALPHA RELIABILITIES FOR THE CAQ-KEYED SUB- 


-SCALES 




Sub-Scale 


No. of 
Items 


1965 Study^ 
(N = 460) 


1973 Study 
(N = 5,297) 


Gen Crse 
Attitude 


8 


.85 


.93 


Ifethod 


8 


.92 


.94 


Content 


8 


.56 


.81 


Interest-Attention 


8 


.89 


.94 


Instructor 


8 


.72 


.82 


Other 


10 


.57 


.75 



^Dick (1965, p. 14) 
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TABLE 6 

OOBmaENT ALFBn RELIABILITIES 
FOR THE CAQ FACIOR-DBRIVED SUB-SCALES 



Factor* 
Scale 


Correaponding 
CAQ Sab-8cale 


Coaponenta 
Reliability 


No. of 
Itena 


Alpha Factor 
ReliabUity 


No. of 
Itema 


I 


Gen.Crae.ACt. 


.96 


U 


.95 


15 


IV 


Mathod 


.94 


9 


.94 


8 


II 


Content 


.83 


6 


.81 


7 


VI 


Int. -Attn. 


.89 


6 


.91 


7 


III 


Inatructor 


•89 


8 


.83 


8 


▼ « VII 


Other 


.59 


7 


.62 


5 



*TWo factora reaaafeUng the '*Other" aub-acale were eoaleaeed for eaae of 
Interpretation. 
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